Unix Shell

Command Line Basics

Prof. Matthew G. Son

University of South Florida

Basic Commands

  • pwd: Print working directory address
pwd
  • ls: List directory contents
ls
ls -a  # Include hidden files
ls -l  # Detailed list view
ls -lh  # human readable format for size
ls -alh # combine above options
  • cd: Change directory
cd /path/to/directory
cd ..  # Go up one level
cd ~ # Go home Directory
  • mkdir: Make (create) a new directory
mkdir myfolder                # myfolder in current directory
mkdir -p folder/subfolder     # create parent folder as needed
mkdir "My Course Materials"  # name with spaces (use quotes)

Special Symbols

  1. . (dot):
    • Represents the current directory.
    • Example: cd . keeps you in the current directory, explicitly specifying the current directory.
  2. .. (double dot):
    • Represents the parent directory (one level up).
    • Example: cd .. moves you up one directory level.
  3. ~ (tilde):
    • Represents the user’s home directory.
    • Example: cd ~ navigates to the home directory.

  1. / (slash):
    • Represents the root directory of the filesystem.
    • In windows it’s similar to C:\
    • Example: cd / navigates to the root directory.
  2. - (hyphen):
    • Represents the previous directory.
    • Example: cd - navigates to the previous directory.
  3. Relative vs. Absolute Paths:
    • Relative paths start from the current directory (e.g., cd ../folder).
    • Absolute paths start from the root directory (e.g., cd /home/user/folder).

Exercise

  1. Use pwd to display your current working directory.
  2. Change working directory to parent directory by one level (use ..). Print the working directory.
  3. Navigate to your home directory using ~ and display the address.
  4. List all files in your home directory, including hidden files.
  5. Create a folder for this classroom (e.g. FIN4770), wherever you’d like to.
  • e.g. If I wanted to create folder on home directory:
    • ~/FIN4770 should be the address

File Management

Create and Delete

  • Create a file:
touch filename.txt # create empty file with .txt extension
  • Remove a file:
rm filename.txt
  • Create a directory:
mkdir directory_name
  • Remove a directory:
rmdir directory_name # works when dir is empty
rm -r directory_name # works when dir is filled

Exercise

  1. In your classroom folder, create a temporary file named temp, verify it exists (ls), then remove it.

  2. Create a temporary directory named tempdir, verify it exists.

  3. Create a temporary file temp2 inside tempdir.

  4. Try to remove the folder temdir with rmdir. What happens?

  5. Remove the temporary directory properly.

Renaming and Moving Files

  • Rename a file:
mv old_name.txt new_name.txt
  • Move a file:
mv file.txt /new/path/

Exercise

  1. Navigate to home folder, using ~.
  2. Navigate back to your classroom FIN4770 folder. Use (-).
  3. Create a file called test.sh then rename it to example.sh.
  4. Make a subfolder shell.
  5. Move example.sh into the shell subdirectory.
  6. Delete example.sh you just created.

Using echo and Pipes

Displaying Text with echo

  • Basic usage:

    echo "Hello, Shell"

  • Redirecting output to a new file:

    echo "This is a line of text" > file.txt

  • Appending output to an (existing) file:

    echo "Yet another line of text" >> file.txt

Combining with Pipes

  • Using echo with wc:

    echo "one two three four" | wc
echo "one two three four" | wc -l # line
echo "one two three four" | wc -w # words
echo "one two three four" | wc -c # characters (+1 newline by default)

Exercise

  1. Write a command using echo to display the text Learning Shell is fun.
  2. Redirect the output of echo "Learning Shell is fun" to a file named Shell_fun.txt.
  3. Use echo and wc to:
    • Count the number of lines in the string Shell pipes are powerful.
    • Count the number of words.
    • Count the number of characters.

Introduction to Shell Text Editors

Shell Text Editors

Two common command-line text editors:

  • nano: Beginner-friendly, simple interface

    • Commands shown at bottom of screen

  • vim: For Pros, Nerds … like me

    • If you want to flex your skills, go for it
    • Make your coding extremely efficient
    • Modal editor (insert mode vs. command mode)

Using nano

  • Open a file:
nano # open a new empty file without filename
nano file.txt # open/create a file with name
  • Basic commands:
    • Ctrl + O: Save file
    • Ctrl + X: Exit editor
    • Ctrl + K: Cut (remove) line

Using vim

  • Open a text file:
vim file.txt
  • Navigation:
    • h, j, k, l: Move left, down, up, right
  • Modes:
    • i: Insert mode (to edit text)
    • Esc: Command mode
  • Basic commands:
    • :q: Quit
    • :q!: Quit without saving
    • :w: Write file
    • :wq: Write and quit

alias

You can create alias that can be used easily. For example, if I wanted to alias my class folder:

# Add to ~/.bashrc (windows git bash users) 
# or ~/.zshrc (macos default)
alias f4770='cd ~/Desktop/FIN4770_class'

After restarting terminal typing f4770 will change directory.

Exercise

  1. Open a text file with nano. Write “Hello world!” and save it as nano_text.txt in shell folder.

  2. Generate a text with vim, write some text, and save as vim_text.txt in shell folder.

  3. (Homework) Generate alias for your class directory. Write text file with exact file location and name:

    • ~/.bashrc (windows) or ~/.zshrc (mac).

Resources for vim

For those who are devoted…

  1. vimtutor: teaches you how to use vim in the command line.

  2. vimHero: Great interactive tutorial for vim

    • https://www.vim-hero.com

  3. vim Adventure: Game that teaches you how to use vim.

    • https://vim-adventures.com

Web API Tools for Data Retrieval

Download data

Downloading Data on current directory

  1. Using curl (general tool):
# Caution: Capital letter "O" for original name. Not zero!
curl -O https://example.com/data.csv
# Different file name
curl -o mydata.csv https://example.com/data.csv
  1. Using wget (specifically for downloads):
wget https://example.com/data.csv

Exercise: Using curl or wget

  1. Navigate to your FIN4770 folder, make a subfolder Data then navigate to it.

  2. Download a sample dataset using either wget or curl.

    • https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv

Note

In case of certification error, try:

curl –ssl-no-revoke -O https://…/iris.csv

Working with Data Files

Viewing (text) Files

  • View all contents of a file:
cat file.txt
  • View the first or last lines:
head file.txt  # First 10 lines
head -n 5 file.txt  # First 5 lines

tail file.txt  # Last 10 lines
tail -n 5 file.txt  # Last 5 lines

Exercise

  1. Browse head of the iris dataset you downloaded.
  2. Browse last 5 lines of iris dataset.

Permissions and Access

Understanding ls -l Output

-rw-r--r--  1 user group  1234 Jan 25 10:30 example.txt

  1. -rw-r--r--: File type and permissions:

    • The first character indicates the type of file:
      • -: Regular file
      • d: Directory
      • l: Symbolic link (symlink)

  1. -rw-r--r--: File type and permissions:
    • The next 9 characters represent the permissions:
      • Split into 3 groups of 3 characters each:
        • Owner (user): First three characters (e.g., rw-)
        • Group: Middle three characters (e.g., r--)
        • Others: Last three characters (e.g., r--)
      • Each group consists of:
        • r: Read permission
        • w: Write permission
        • x: Execute permission
        • -: No permission
    • Example:
      • rw-: Read and write, but no execute
      • r--: Read-only
      • rwx: Read, write, and execute

-rw-r--r--  1 user group  1234 Jan 25 10:30 example.txt

  1. 1: Number of hard links to the file or directory.

    • Multiple addresses for the same existing file
    • Doesn’t count symlinks

  2. user: The owner of the file.

  3. group: The group associated with the file.

  4. 1234: The size of the file in bytes.

  5. Jan 25 10:30: The last modification date and time.

  6. example.txt: The name of the file or directory.

Detailed Example

Output:

drwxr-xr-x  2 user group   4096 Jan 25 12:00 folder
-rw-r--r--  1 user group   1234 Jan 25 10:30 file.txt
lrwxrwxrwx  1 user group     20 Jan 25 10:45 link -> /path/to/target

Explanation of drwxr-xr-x for folder:

  • d: Directory.
  • rwx: Owner (user) has read, write, and execute permissions.
  • r-x: Group has read and execute permissions.
  • r-x: Others have read and execute permissions.

Exercise

  1. Check the permissions of iris file you downloaded.

    • What is the file type?
    • What is the access permission for owner (user)?
    • What is the access permission for group?
    • What is the access permission for others?

Changing Permissions

  • Change permissions:
chmod 756 file.txt  # Set read-write-execute
  • Each digit corresponds to user, group, and others
    • Read (r): 4
    • Write (w): 2
    • Execute (x): 1
    • No permission (-): 0
  • 756 breaks down as:
    • User (7): rwx (4 + 2 + 1 = 7)
    • Group (5): r-x (4 + 0 + 1 = 5)
    • Others (6): rw- (4 + 2 + 0 = 6)

Exercise

  1. Create a file and view its permissions with ls -l.
  2. Change the permissions of the file to rwxr---w- using chmod.
  3. Interpret chmod 600 file.txt.

Writing and Running Script Files

How to Write a Script File

  1. Create a Script File:

    • Use a text editor like nano or vim to create a script file:

      nano myscript.sh
  2. Add Commands to the Script:

    • Begin with a shebang to specify the shell interpreter:

      #!/bin/bash

    • Add commands to the script:

      echo "Hello, Shell!"
echo "My current shell is $SHELL"
date
pwd
  3. Save the File:
    • Save and exit the editor (Ctrl + O and Ctrl + X in nano).

How to Run a Script File

  1. Make the Script Executable:

    • Grant execute permissions to the script using chmod:

      chmod 755 myscript.sh
  2. Run the Script:

    • Execute the script by specifying its path:

      ./myscript.sh
  3. Alternative Method:

    • Run the script without changing permissions by invoking the shell directly:

      bash myscript.sh

Exercise: Writing and Running Scripts

  1. Create a script file named greet.sh that:

    • Prints “Welcome to Shell scripting.”.
    • Displays the current date and time.

  2. Add execute permissions.

  3. Run the script.

Working with Processes

Viewing Processes

  • List running processes:
ps
ps u  # only current user
  • Interactive process viewer:
# q to quit
top
htop

Managing Processes

  • Kill a process:
kill PID
  • Background and foreground processes:
command &  # Run in background, PID is printed
fg  # Bring to foreground

Exercise

  1. Use ps u to display your running processes.
  2. Run sleep 5. What does it do?
  3. Start a background process using sleep 100 & and view it with ps u or jobs.
  4. Kill above sleep process with kill <PID>.

Command-Line Productivity

History and Shortcuts

  • View command history:

    history
    • Displays a list of previously executed commands.

  • Repeat a command:

    !!  # Repeat the last command

  • Run a specific command from history:

    !<number>  # Replace <number> with the history entry number

  • Clear the command history:

    history -c # bash
history -p # zsh

Exercise

  1. View your command history and re-run the last three commands.
  2. Use !! to repeat the last command.
  3. Use !<number> to execute a specific command from the history list.
  4. Experiment with Ctrl + r and type keyword to find a previously used command.
  5. Clear your command history and verify it is empty with history.

Essential Shortcuts

  • Ctrl + A: Move to the beginning of the line
  • Ctrl + E: Move to the end of the line

Line Editing

  • Ctrl + U: Delete everything before the cursor
  • Ctrl + K: Delete everything after the cursor
  • Ctrl + W: Delete the word before the cursor

Command History

  • Ctrl + R: Search your command history interactively
    • Press Ctrl + R repeatedly to cycle through matches

Miscellaneous

  • Ctrl + L: Clear the terminal screen (history preserved)
  • clear: Clear and remove history